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© Programmable on-focal plane signal processor. 



© A programmable on-focal plane signal processor 
(20, 21) having analog to digital conversion (20), 
nonuniformity correction, gamma spike compensa- 
tion all digitally performed on a single silicon sub- 
strate (18) inside a dewar. The architecture supports 
time delay integration and spatial filtering to increase 
the signal to noise ratio of the focal plane array (1) 



data. The processor has programmable coefficients 
which may be changed while the system is operat- 
ing or eliminated. The architecture provides in- 
creased signal noise ratio for infrared data and de- 
creases the output data bandwidth from the infrared 
focal plane array (1) by two orders of magnitude. 
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BACKGROUND OF THE INVENTION 



1 . Technical Field 



The invention relates generally to signal pro- 
cessing and more particularly, to space-borne sig- 
nal processing for use at low temperatures and in a 
radiation environment. 

2. Background Art 



The electrons emitted by the detectors are . cap- 
tured beneath each pixel and the quantity of elec- 
trons captured is directly proportional to the inten- 
sity of the objects within the immediate field of 

5 view (IFOV) of the pixel. A typical focal plane array 
is composed of 16,384 pixels arranged as a matrix 
of 128 by 128 elements. The data from ail pixels in 
the array can be read as one snapshot of the 
scene. This snapshot is called a frame of IR data. 

io The signals from the pixels of the FPA 16 are 



Space based threat warning systems use in- 
frared focal plane arrays to detect potential missile 
threats. However, infrared focal plane arrays (FPAs) 
which operate in space have two inherent limita- 
tions. They must function in a cryogenically cooled 
environment (i.e. in a dewar) and their output must 
be corrected to compensate for errors induced by 
incident gamma rays. Currently, errors caused by 
gamma radiation, known as gamma spikes, are 
corrected using analog processing in a dewar or by 
digital or analog processing outside the dewar. 
Using analog processing inside the dewar or digital 
processing outside the dewar increases the amount 
of noise introduced into the FPA data. 

A typical focal plane signal processor is shown 
in Figure 13. Figure 13 shows a typical infrared 
missile warning system with a staring sensor. A 
lens 12 is used to focus the IR energy from a 
portion of the sky/ground scene through an optical 
filter 14 and onto a focal plane array 16. The lens 
typically has a 60 • x 60 • field of view (FOV) and 
allows the sensor to collect IR energy from or 
"stare at" a portion of the sky/ground scene. This 
type of system is known as a staring system. 
Another category of infrared missile warning sys- 
tem uses a scanning sensor. In the scanning sys- 
tem the sensor is a line array (rather than a rectan- 
gular FPA in a staring system) and the optics scan 
the scene through the line array. 

The filter 14 can a color wheel. Alternatively, 
the color wheel may be replaced with an acousto- 
optic tunable filter (AOTF). The purpose of an op- 
tical filter is to limit the thermal energy which 
.passes to the FPA. Targets have signatures with 
large amounts of thermal energy in very specific 
frequency bands. The optical filters are tuned to 
allow only energy in the target specific frequency 
bands to pass. Therefore, targets imaged by the 
FPA will have a greater signal to noise ratio after 
optical filtering. 

The focal point array 16 is a matrix of individ- 
ual detector elements or pixels, which emit elec- 
trons when struck by incident photons. The term 
pixel is an abbreviation for picture element and is 
the smallest uniquely definable element in the 
scene. The value of each pixel represents the 
amount of infrared energy incident on a detector. 



amplified by amplifier 18 and converted to a digital 
form in A/D converter 20. The data are then pro- 
cessed to compensate for nonuniformities in the 
FPA which were introduced when the array was 

15 manufactured. The compensated data are spatially 
filtered to increase the signal to noise level of 
potential targets in the data. After spatial filtering, 
the data may be spectrally filtered and detected. 
All this is done in signal processor 22. Any de- 

20 tected targets are sent to the data processor for 
further processing. An inertia! navigation system 
(INS) 26 outputs data to the data processor 24 to 
aid in tracking any detected threats. 

What is needed is a focal plane processor 

25 having low noise with correction for gamma errors 
and able to operate at or below liquid nitrogen 
temperatures and be radiation hardened. This can 
be done by digitally processing the FPA data in- 
side the dewar to reduce the noise and provide 

30 added resolution. The added resolution yields sig- 
nificantly better target detection and tracking per- 
formance. 



.35 



SUMMARY OF THE INVENTION 



The signal processor of the present invention 
performs analog to digital conversion, nonunifor- 
mity correction and gamma spike compensation, all 
digitally, on a single silicon substrate inside a dew- 

40 ar. The signal processor supports additional al- 
gorithms such as time delay integration and spatial 
filtering to increase the signal to noise ratio of the 
FPA data. The processor is programmable and the 
coefficients used by the signal processor algorithm 

45 can be changed while the system is operating or 
eliminated if not required. The processor increases 
signal to noise ratio for targets in IR data and 
decreases the output data bandwidth from the in- 
frared FPA by two orders of magnitude. 

50 

BRIEF DESCRIPTION OF THE DRAWINGS 

Figure 1 is a block diagram of the focal 
plane array according to the 
55 present invention. 

Figure 2 is a block diagram of the process 
flow for time dependent process- 
ing according to the present in- 
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vention. 

Figure 3 is a schematic diagram of the 
gamma circumvention circuit of 
the present invention. 
Figure 4 is a dead pixel detection imple- 
mentation schematic of the 
present invention. 
Figure 5 is a schematic of the implementa- 
tion for dead pixel correction cir- 
cuit of the present invention. 
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In addition to the radiation hardness inherent in 
the LNCMOS process, several standard fault toler- 
ance techniques have been incorporated in the 
processor. Parity bits are used on all address and 
data buses to detect parity errors. If an error is 
detected, each device has a programmable retry 
capability to correct the error. All devices are man- 
ufactured using Level Sensitive Scan Design 
(LSSD) techniques for increased testability. Built-in- 
test (BIT) capabilities are included in each device 



Figure 6 shows an implementation of non- 
uniformity compensation accord- 
ing to the present invention. 

Figure 7 is a schematic of the frame sub- 
straction circuit of the present in- 
vention. 

Figure 8 is a schematic of the time delay 
and integration implementation of 
the present invention. 

Figure 9 is a schematic of the spatial filter- 
ing according to the present in- 
vention. 

Figure 10 is a schematic of the Mean Cal- 
culation of the present invention. 

Figure 11 is a schematic of a standard de- 
viation calculation of the present 
invention. 

Figure 12 is a schematic for implementation 
of threshold exceedance within the 
present invention. 

Figure 13 is a block diagram of a conven- 
tional infrared focal plane array. 

DESCRIPTION OF THE PREFERRED EMBODI- 
MENT OF THE PRESENT INVENTION 

Signal Processor Overview 

The signal processor described in this patent 
application provides a flexible, radiation hardened, 
high performance solution to processing IR threat 
warning data. By processing all data digitally within 
the dewar, the processor yields inherently better 
performance than existing systems since the con- 
tribution of analog noise sources is eliminated. As 
an added benefit, the processor operates at twice 
the clock frequency of room temperature CMOS 
systems. Altogether, these advantages combine to 
produce a compact, cost effective architecture 
which processes IR threat warning data in real 
time. 

To counter the effects of radiation, all logic and 
memory devices is made using radiation hardened 
CMOS. Furthermore, the standard radiation har- 
dened process will be combined with a technique 
for liquid nitrogen CMOS (LNCMOS) processing to 
provide the benefits of both low temperature and 
radiation hardness. 



and an on-chip monitor (OCM) is used as a test 
interface. All memory components have built-in er- 
ror correcting codes (ECC) and column sparing to 
correct multiple bit errors. 
;s The first subsection of this disclosure dis- 

cusses the systems requirements for the signal 
processor. The second subsection details the ar- 
chitecture design. 

20 Programmable On-focal Plane Signal Processor 

A two stage dewar or cryocooler is required for 
this task; the A/D converter must operate at 4*K 
and the CMOS VLSI chip on silicon (VCOS) must 

25 operate at 77 W K. Figure 1 shows the focal plane 
array (1), and its multiplexing circuitry (2) on one 
silicon substrate (19) on the cold finger attached to 
the first stage of the Dewar. The FPA is assumed 
to be as large as 256 x 256 pixels with a maximum 

30 frame rate of 400 frames per second (fps). 

The analog output from the FPA is multiplexed 
by the analog multiplexer (2) and transferred via a 
single serial wire to the analog to digital (A/D) 
converter (3). The A/D is bonded to a separate 

35 silicon substrate (20) but resides on the same cold 
finger as the FPA substrate (19) or can reside 
inside the dewar. The A/D converter must perform 
26.2 million conversions per second to meet the 
output of the analog multiplexer and should have 

40 16 bit accuracy. Several A/D converters have these 
characteristics. For example, the Hypress (TM) A/D 
converter uses a flux quantizing technique and is 
manufactured from superconducting interference 
devices (SQUIDs) in Niobium Nitride technology. 

45 Currently, Hypress manufactures A/D converters 
which provide 16-bit bits of resolution operating at 
200 MHz. The format for the data will be frame 
interleaved; the four samples for the first pixel in 
four successive frames will be output first followed 

so by the four samples for the second pixel, etc.. 
These data shall be passed bit-wise parallel to the 
second stage of the dewar. 

The second stage of the dewar houses a single 
multi-chip-module (21) containing the devices re- 

55 quired to perform time dependent processing. 
These devices will process the input data, detect 
targets and output a list of potential targets, their 
intensities and locations for further processing by a 
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processor external to the dewar. All processing and 
memory devices, with the exception of the process 
controller (17) and the output circuit (16) devices, 
will be mounted on a 12 chip VLSI chip-on-silicon 
(VCOS) substrate. The process controller device 
will provide timing and control signals to the de- 
vices in the 4 • K dewar stage as well as to devices 
mounted in the VCOS. The output converter device 
converts the CMOS output from the processing 
module to fiber optic format. The output converter 



at 100 MHz. 

The Output Convert device (16) converts the 
32-bit, parallel, CMOS output from the Threshold 
device to an error control coded serial stream for 

5 the filter optic interface. To transmit the threshold 
data, the Output Convert device contains an en- 
coder, a parallel/serial converter and drive circuitry 
for the laser transmitter. The Output Convert device 
also receives instructions from an external mission 

io data processor and forwards these instructions to 



device must remain off-VCOS because of the 
bonding requirements of the fiber optic cable and 
the device's incompatibility with the VCOS cover. 

All memory buffers and data bandwidths used 
in this architecture are sized to accommodate a 
256 x 256 pixel FPA sampled at 400 fps. However, 
larger FPAs and faster frame rate can be used if 
the memory density is increased. Sufficient band- 
width exists at the VCOS and device boundaries to 
accommodate larger arrays and higher frame rates. 

To the greatest extent possible, control switch- 
es and programmable variables have been includ- 
ed to provide increased flexibility. Each processing 
stage can be bypassed and intermediate results 
can be analyzed for accuracy and debug purposes. 
Functions have been partitioned onto chips in a 
logical manner such that alternative functions can 
be added to, or exchanged with current functions 
as needed without redesign of the system. 

This architecture also allows a high degree of 
parallelism. Both multiple sensors and multiple col- 
ors can be accommodated through channelized 
processing. Multi-sensor fusion or multi-spectral 
processing can be accomplished by combining the 
thresholded data in an external processor. 

Ail logic devices are manufactured using the 
LNCMOS process and will be bonded to the silicon 
substrate using a C4-process. The output conver- 
sion and the process controller devices are also 
manufacturer using the LNCMOS process but are 
not bonded to the silicon substrate. 

Figure 1 shows the placement of the 12 logic 
chips in the VCOS package. The data input to the 
Correct Circuits device (8) in the VCOS is 16-bits 
wide. The Correct Circuits device in Figure 1 im- 
plements gamma circumvention, dead pixel detec- 
tion, dead pixel correction, nonuniformity correc- 
tion, and TDI. This device interfaces with several 
memory devices (4, 5, 12, 13), and sends its re- 
sults to the Spatial Filter devices (9, 10). The 
Spatial Filter devices convert the integer data from 
the Correct Circuits device to floating point, per- 
form a separable filter on the data and pass the 
spatially filtered data to the Threshold Mean and 
Standard Deviation devices (10, 14, 15, 6, 7). Spa- 
tial filtering and thresholding are performed using 
the IEEE 32-bit floating point format. AH logic chips 
in the VCOS have 0.5u. feature size, and operate 



the Process Controller (17) device. To receive 
these serial data, the Output Convert device con- 
tains a laser receiver, a serial to parallel converter 
and a decoder. The laser used in the Output Con- 

15 vert device will be chosen to minimize the amount 
of heat generated to reduce the impact on the 
thermal capacity of the cryocooler. The optical fiber 
will be attached directly to the chip. 

The Process Controller device will generate the 

20 system clock for the FPA assembly (19), the A/D 
converter (3), the VCOS package components and 
the Output Convert device. The Process Controller 
device (16) will also generate the control signals 
and provide coefficients and processing variables 

25 as needed to the devices in the VCOS package. 
The Process Controller device collects operating 
status and error information. This information will 
be stored in the Process Controller device and will 
be transmitted to the external processor via a fiber 

30 optic link. 

The VCOS module, the Output Converter de- 
vice and the Process Controller device will be 
mounted on a Kevlar or epoxy/glass common mod- 
ule. The entire module will be encapsulated in 

35 Parylene and this module will be mounted inside 
the dewar. 

The fiber optic interface carries the output data 
and status information to an external Mission Data 
Processor and carries control information from the 

40 MDP for input to the Process Controller device. 
One end of an optical fiber will be directly attached 
to the Output Convert chip shown in Figure 1. The 
other end of the fiber will be attached to a connec- 
tor on the cryocooler housing. Several clamps will 

45 secure the optical fiber within the dewar to mini- 
mize flexing. A passive coupler brings the fiber 
through the cryocooler housing. This minimizes the 
number of connectors and reduces the system's 
mechanical instability due to shock and vibration. 

so The passive coupler should maintain the cryocooler 
seal and provide a flexible mating attachment. The 
proposed architecture does not require a fiber op- 
tics link. Data rates from the threshold algorithm 
are of limited bandwidth and, alternately, a serial 

55 wire interface could be used. 
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Algorithm Implementation 



Dead Pixel Processing 



This section details the implementation of the 
time dependent processing algorithms to be later 
discussed. The implementation of each algorithm is 
discussed in the order in which processing occurs. 
Figure 2 shows a high level algorithm flow for this 
signal processor. 

Gamma Circumvention 



The two components of dead pixel processing 
are dead pixel detection and dead pixel correction. 
5 These are discussed in the following subsections. 

Dead Pixel Detection 

Dead pixel detection consists of checking the 
70 data from each pixel for a value of zero or 65,535. 



Gamma circumvention processing inputs four 
oversampled data values for each pixel and outputs 
a single value which represents the average of the 
smallest two input data values. In this manner, 
gamma spikes, characterized as random, high in- 
tensity values, will be removed from the pixel data. 

Figure 3 shows the hardware implementation 
for gamma circumvention as a three stage pipeline. 
The first two stages contain 16-bit comparators, 32 
and 34 respectively, that find the smallest two input 
data samples in the four oversampled points for 
each pixel. The last stage contains a 16-bit adder 
36 which adds the smallest two values. The output 
of the adder is shifted right one bit to divide the 
resulting 17 bit number by two. The average of the 
two smallest values is output from this circuit. In 
addition to filtering gamma spikes from the data, 
this circuitry also reduces the sensor data band- 
width to 100 fps. 

As the four data samples pass through the 
comparator function, each new sample is com- 
pared with a previously stored value. Initially, this 
value is 65,535 (the largest value of a 16-bit A/D 
converter). The smaller of the two values is stored 
in the Smallest Value register as MIN and the 
greater value is passed to the next comparator as 
min. The second comparator repeats this process 
and stores the smaller value, MIN, in its Smallest 
Value register. The second stage Smallest Value 
register is also initialized to all ones. The greater of 
the two values, min\ is not used. After all four 
samples have passed through the comparators, the 
values contained in the Smallest Value registers 
are the two smallest values in the four value se- 
quence. These two values are then averaged and 
output as the filtered data sample. The Smallest 
Value registers in each comparator stage are reset, 
via the Reset Signal 38, to contain all ones in 
preparation for the next data set. The Process 
Control device controls the reset signal and pro- 
vides the pixel clock signal. 

Responsivity Correction 

Responsivity correction is implemented in two 
stages: dead pixel processing and nonuniformity 
correction. 



If either of these conditions are met, the pixel is 
listed as a bad pixel and the location of this pixel is 
passed to the dead pixel list. 

Figure 4 shows the dead pixel detection cir- 

75 cuitry. All sixteen bits for each pixel value are 
simultaneously input to 16 input NAND 40 and OR 
42 gates. If the output of either the NAND or OR 
gates are zero, it implies that the pixel's value was 
either 65,536 or zero, respectively and the pixel is 

20 assumed to be dead. The outputs from the NAND 
and OR gates are input to a two input AND gate 
44. The AND gate will output a zero if the pixel is 
dead and this value will cause the latch 46 to hold 
the value of the 16-bit binary counter. This signal is 

25 also used as the Dead Pixel Indicator signal in the 
dead pixel correction circuitry. The counter con- 
tains the location of the pixel within a frame of data. 

The counter is reset at the beginning of each 
frame and is clocked by the Pixel Clock signal. The 

30 pixel clock increments the counter each time a new 
pixel value is input to the dead pixel detection 
circuit. The output of the counter is input to the 16 
bit latch. When the latch is strobed by the AND 
gate output, the position of the pixel relative to the 

35 first pixel in the frame is stored in the latch and 
passed to the Process Control device for inclusion 
into the dead pixel list. 



40 



Dead Pixel Correction 



The dead pixel correction algorithm replaces 
the dead pixel with the average of its left (previous 
pixel) and right (next pixel) neighbors. Figure 5 
shows the dead pixel correction circuitry. 

45 The inputs to the circuitry are the input data 

and also the Dead Pixel Indicator signal from the 
Dead Pixel Detection circuit. When the Dead Pixel 
Indicator signal goes low (signifying that the current 
pixel is dead), the contents of the registers 50, 52, 

so 54 in the Figure 5 contain the value of the current 
pixel and the two previous pixels. To implement the 
algorithm requires that the previous pixel, the cur- 
rent pixel and the next pixel be captured in the 
three registers. To capture the values of the left 

55 and right neighbors in the first and third registers, 
however, requires that the Dead Pixel Indicator 
signal be delayed by one clock cycle. The Dead 
Pixel Indicator signal is latched in a one bit flip-flop 
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56 and when the next clock cycle is active, the 
values of the left and right neighbors are summed, 
averaged and output in place of the dead pixel. If 
the current pixel is not dead, this circuit passes the 
pixel value unchanged. 

Nonuniformity Correction 

Nonuniformity correction uses a single pole, 
temporal. Infinite Impulse Response (IIR) filter for 



Frame Subtraction 

Frame subtraction subtracts a pixel's value in 
the previous frame from the pixel's value in the 

5 current frame. Figure 7 shows the implementation 
of frame subtraction. The previous frame's data is 
stored in the Frame Store RAM 70 which is a dual 
ported memory. To subtract the previous pixel val- 
ues from the current pixel values, data from the 

io previous frame is retrieved from the Frame Store 



each pixel. Figure 6 shows the design for non- 
uniformity correction. The input data is routed to 
two paths: the gain correction path and the IIR 
filterpath. Figure 6 shows the gain correction path 
as the upper path 60 and the IIR filter path as the 
lower path 62. The output of the IIR filter path is 
subtracted from the gain correction path using the 
negate and adder blocks 64, 66 at the bottom of 
Figure 6. The resulting data is input to the Frame 
Subtraction processing stage. 

The input data is corrected for gain nonunifor- 
mities in the gain correction path. Each pixel value 
is multiplied by a 16-bit gain constant. The gain 
constants are stored in the Gain RAM 68 shown in 
the upper left portion of Figure 6. The gain cor- 
rected input data is input to one side of the adder 
and the IIR filter output is negated and input to the 
other side of the adder. The NUC value for the 
pixel is then calculated by adding these two inputs. 

In the IIR filter path, the input data is first 
shifted by a programmable filter weight which has 
a value between 0 and 15, inclusive. This value is 
applied to the pixel data for all pixels within a frame 
and is represented as the four bit scaling constant, 
£, shown in Figure 6. The scaling constant is 
maintained by the Process Control device and its 
value is set during calibration but can be modified 
during operation. 

A factor related to the previous value for each 
pixel is then added to the scaled input data. This 
factor is calculated by multiplying a filter weight, a, 
with the previous output value for the current pixel. 
The filter weight is determined prior to operation 
and is loaded into the Correction Circuits device by 
the Process Control device. Figure 6 shows that 
the previous output value for the current pixel is 
stored as a 16-bit value in the Filter History RAM 
69 which is a dual ported memory. 

The result of the ADD operation is negated and 
passed to the ADDer and is also written to the 
Filter History RAM. The Filter History RAM is also 
read as one of the two inputs to the multiplier. 
Therefore, the RAM must be accessed twice for 
each pixel. For a pixel rate of 26.2 M pix- 
els/second, the Filter History RAM must be acces- 
sed at 52.4 MHz. This is easily accomplished with 
devices produced with currently available LNCMOS 
processes. 



RAM, two's complemented, and added to the 
pixel's current value. The result is passed to the 
Time Delay and Integration algorithm stage. 

To avoid overwriting the previous pixel value 
J5 with the current value, the Frame Store FIAM is 
written by a register that holds the current input 
data value until the previous data value has been 
read. 

20 Time Delay and Integration 

Figure 8 shows that the Time Delay and In- 
tegration (TDI) stage is implemented as a sum and 
dump integrator. A number of successive frames, 

25 N, are averaged and the result is output to the 
Spatial Filter device. The number of frames in- 
tegrated, N, is also the scaling constant, and is 
supplied by the Process Control device. In this 
design, N can have a value of 0, 2, 4, or 8. 

30 The Frame History RAM 80 is a dual ported 

RAM and is written with the result of the add. The 
Frame History RAM is read as one of the two 
inputs to the ADDer. The Frame History RAM loca- 
tion corresponding to a particular pixel must be 

35 read before it is written with the current sum. After 
the sum of N frames is calculated, the Frame 
History RAM is reinitialized to zero in preparation 
for the next frame. 

40 Two Dimensional Convolution (Spatial Filtering) 

Two dimensional convolution is implemented 
as a symmetric, separable, spatial filter. A separa- 
ble filter processes data in two passes; first along 

45 the rows of the frame and then, after corner turning 
the intermediate data, along the columns of the 
frame. All filter weights are programmable and are 
loaded into the Spatial Filter devices by the Pro- 
cess Control device. The weights are stored in a 

so bank of three registers and can be changed during 
operation. Two different filter sizes are available: 3 
x 3 and 5x5 pixels. The filter size should match 
the size of the expected targets. The choice of filter 
size is controlled by the Process Control device. 

55 Figure 9 shows that two devices are required 

to implement the spatial filter. The first device 90 
performs the first pass of the convolution and part 
of the corner turning process. The second device 
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92 completes the comer turning process and the 
second pass of the filter. 

Figure 9 shows that the input data is first 
converted from 16-bit integer to 32-bit floating 
point. For a 5 x 5 pixel filter, the input data is 
sequentially loaded into a bank of registers 94. As 
a new data value enters the register bank, the 
oldest data value is discarded. If a 3 x 3 pixel filter 
is selected, only the two most recent data values 
are sequenced through the register bank. The fol- 



Statistics Generation 

The Contrast Box algorithm requires calculation 
of the mean and variance of the target and back- 

5 ground boxes, u ( , u b , of and a§ respectively. To 
compute these, our implementation presented in 
this report uses two parallel data paths; one path 
computes the mean values of the target and back- 
ground boxes, and the second path computes the 

10 variances. The size of the target box is 3 x 3 pixels 



lowing discussion focuses on the implementation of 
a 5 x 5 pixel filter. 

The filter is symmetric, therefore, the first and 
fifth data values and the second and fourth data 
values will be multiplied by the same filter coeffi- 
cient. To take advantage of the symmetry and to 
reduce the number of 32-bit multipliers, the first 
and fifth values are added and then multiplied by a 
filter coefficient. In a similar fashion, the second 
and fourth values are added and the sum multiplied 
by a second filter coefficient. The third or "center" 
pixel is multiplied by a third coefficient and added 
to the outputs of the other multipliers. The output of 
the adders is the output of the first pass Finite 
Impulse Response (FIR) filter. The resulting data 
are sequentially input to four 256 x 32-bit FIFOs 
which perform the corner turn. The first two FIFOs 
are located on the first spatial filter device and 
second two FIFOs are located on the second spa- 
tial filter device. 

The second spatial filter device applies the 
filter coefficients of the second pass to the corner 
turned data. In a manner identical to that used on 
the first spatial filter device, the coefficients are 
multiplied with the input data and the results are 
summed using two adders. The resulting data is 
the spatially filtered imagery and is output to the 
threshold exceedance processing stage. 

Threshold Exceedance 

The Contrast Box algorithm outlined in Section 
3.5 is used to calculate threshold exceedances. 
This algorithm is implemented in two stages: statis- 
tics generation and thresholding. Statistics genera- 
tion calculates the statistics of the target box and 
the background box. Thresholding compares the 
statistics of the target and the background boxes 
and determines if the boxes are statistically dif- 
ferent. The measure of this difference is the con- 
trast metric, C. If C is greater than a threshold, K, a 
detection is declared and the detection's location 
and statistics are passed to the Clump Processing 
algorithm stage. 



and the size of the background box can be either 5 
x 5 or 7 x 7 pixels. 

Both paths calculate the statistics for their tar- 
get and background boxes simultaneously and both 

75 paths use a separable scheme. As in the spatial 
filter implementation, a separable filter processes 
the rows of a matrix first, and outputs intermediate 
values which are then processed by columns. 

For the mean value path, the sum of the pixel 

20 values are first calculated along the rows of the 
input matrix. This scheme outputs a value for each 
target box pixel which is the sum of the center 
pixel and its left and right neighbors. Simultaneous- 
ly, the mean value path outputs the sum of the 

25 center pixel and its four or six nearest neighbors in 
the same row for the background boxes which are 
5 x 5 or 7 x 7 pixels in extent, respectively. The 
data from the first pass is then corner turned and 
input to the second pass of the filter. 

30 The second pass is executed by the in the 

mean value second device. The output from the 
second device is the sum of the energies along the 
columns of the output data's first pass. This is the 
sum of the energies around the center pixel in the 

35 target and background boxes. The outputs from the 
mean value path are input to the Threshold device. 

Figure 10 shows the data flow of the mean 
value path. The separable sum is executed by two 
devices: Mean Calculation device #1 100 and Mean 

40 Calculation device #2 102. Mean Calculation device 
#1 receives the pixel clock signal from the Process 
Control device and data from Spatial Filter device 
#2. The data are first input to a six stage FIFO. The 
center three pixel values in the FIFO are summed 

45 to yield the first pass (row) output for the target 
box. The value of the other four pixels in the FIFO 
are also summed and added to the sum of the 
center three pixels. This yields the first pass sum 
for the background box. The first pass results are 

so then input to two separate FIFO stages: one stage 
for the target box and a second stage for the 
background box. The target box FIFO stage is a 
two stage FIFO which corner turns the first pass 
target box output. The background box FIFO stage 

55 is a six stage FIFO which corner turns the first pass 
background box output. By summing the output 
from the FIFO stages, the center pixel value is 
replaced with the sum of its value and the value of 



7 



13 



EP 0 651 566 A1 



14 



its eight, 24, or 48 neighbors for the 3 x 3, 5 x 5 or 
7x7 pixel box sizes, respectively. The outputs of 
the mean value path are u t and u b and these are 
input to the Threshold device. 

The standard deviation path uses the squared 
power of each pixel versus the pixel values used in 
the mean path. Figure 1 1 shows that the first pass 
of the standard deviation is completed in Standard 
Deviation device #1 110 and the corner turn and 
second pass is completed in Standard Deviation 
device #2 112. The output of the Standard De- 
viation device #2 is not the standard deviation but 
the sum of the squared powers in the neighbor of 
the target and background boxes. This sum is 
output to the Threshold device which computes the 
variances and the contrast metric. 

Both paths will use identical chips. The only 
difference between the two paths is that the first 
device in the standard deviation path uses an addi- 
tional multiplier to square the input data. This multi- 
plier would be bypassed in Mean Calculation de- 
vice #1 . 

Thresholding 

Figure 12 shows the Threshold Detection de- 
vice 1 20 which receives input from the mean value 
and standard deviation paths and the threshold 
constant, K, from the Process Control device. The 
variance for the target and background boxes is 
computed in the upper right corner of Figure 12. 
The sum of the squared powers for the target and 
background boxes is received from Standard De- 
viation device #2. To compute the variances, the 
mean of the target and background boxes are 
squared and subtracted from the sum of the 
squared powers. This yields the variance o\ and o\ 

The contrast metric, C, is the ratio of the target 
box power to the background box power. The total 
power in target box is the sum of the mean target 
power and the variance of the target energy. To 
calculate the average energy distributed to the 
target alone, the average background energy, u b , is 
subtracted from the mean target energy u t . This 
difference is squared to yield the mean target 
power. Figure 12 shows the mean target power 
term as (u t - u b ) 2 . The mean target power is added 
to the variance to yield the total power in the target 
box. Figure 12 shows this term as the A input in 
the compare logic. 

The background power is the sum of the aver- 
age background energy squared and the back- 
ground variance. This term is calculated by the 
adder in the upper right corner of Figure 12. The 
background power is multiplied by the threshold 
constant, K, and their product is the B input of the 
compare logic. If the target power is greater than 



the background power/threshold product, a detec- 
tion results and the statistics of the target and its 
background are transmitted to the clump process- 
ing stage. 

5 While the invention has been particularly 

shown to describe with reference to the preferred 
embodiment, will be understood that various 
changes of form and details maybe made therein 
to provide the present invention in settings in al- 

w ternate embodiments different from that described 
without departing from the spirit and scope of the 
invention. 

Claims 

75 

1. A focal plane array (1) processor for infrared 
signal processing comprising: 

a focal plane array (1) for receiving infrared 
signals; 

20 an analog multiplexer (2) for receiving the data 

from the focal plane array (1) and outputting a 
signal; 

an analog to digital converter (3) for digitizing 
the multiplexed signal from the focal plane 

25 array (1); 

the focal plane array (1 ), analog multiplexer (2) 
and analog to digital converter (3) operating at 
or below liquid nitrogen temperatures; and 
a time dependent signal processor (21) formed 

30 on a single substrate (18), receiving the digitiz- 

ed output from the focal plane array (1) and 
outputting a signal indicative of the received 
infrared signal that is corrected for nonunifor- 
mities and gamma spikes occurring in the fo- 

35 cal plane array (1). 

2. The focal plane array (1) processor of claim 1, 
wherein 

the focal plane array (1), analog multiplexer (2) 
40 and analog to digital converter (3) operate at 

4«K. 

3. The focal plane array (1) processor of claim 1 
or 2, wherein 

45 the time dependent signal processor (21) in 

addition comprises responsivity correction, 
frame subtraction, spatial filtering and thresh- 
old adjustment. 

so 4. The focal plane array (1) processor of claim 3, 
wherein 

the responsivity correction includes dead pixel 
detection and dead pixel correction. 

55 5. A focal plane array (1) processor for infrared 
signal processing comprising: 
a focal plane array (1) for receiving infrared 
signals as data; 
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an analog multiplexer (2) for receiving the data 
from the focal plane array (1) and outputting a 
signal; 

an analog to digital converter (3) for digitizing 

the multiplexed signal from the focal plane 5 

array (1); 

the focal plane array (1), analog multiplexer (2) 
and analog to digital converter (3) operating at 
4*K;and 

a time dependent signal processor (21) formed 10 
on a single substrate (18), receiving the digitiz- 
ed output from the focal plane array (1) and 
outputting a signal indicative of the received 
infrared signal said time dependent signal pro- 
cessor (21 ) further comprising: 15 
gamma circumvention circuitry for eliminating 
gamma spikes; 

responsivity correction means for detecting 
and correcting for dead pixels, said respon- 
sivity correction means also providing gain uni- 20 
formity of the digitized signal; 
frame subtraction means for signal enhance- 
ment; 

time delay integration means for signal 
smoothing; 25 
spatial filtering for tuning the digital signal; and 
threshold leveling for generating a desired sig- 
nal level. 



6. A system for observing space-borne missile 30 
threats receiving infrared signals, the signals 
processed by a focal plane array (1 ) processor 
comprising: 

a focal plane array (1) for receiving infrared 
signals; 35 
an analog multiplexer (2) for receiving the data 
from the focal plane array (1) and outputting a 
signal; 

an analog to digital converter (3) for digitizing 

the multiplexed signal from the focal plane 40 

array (1); 

the focal plane array (1), analog multiplexer (2) 
and analog to digital converter (3) operating at 
or below liquid nitrogen temperatures; and 
a time dependent signal processor (21 ) formed 45 
on a single substrate (1 8), receiving the digitiz- 
ed output from the focal plane array (1) and 
outputting a signal indicative of the received 
infrared signal that is corrected for nonunifor- 
mities and gamma spikes occurring in the fo- 50 
cal plane array (1 ). 
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Figure I. layout of Ihc Focal Plane Array Processor 
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Figure 6. Implementation of Nonuniformity Compensation 
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Figure 7. Implementation of Frame Subtraction 



MPUTDATA 



SCALING 



CONSTANT 



18 



/ 

/If 



AOORC53 OCNCRATOR 



PIXEL 



BINARY COUNTER 



AOORC5* 



TDI 
HISTORY 
RAM 



V 



ADOER 



f 



COUNT -N 



h 

03 



1« 



TR INSTATE 



I ovra/r DATA 



rcset 
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13 



EP 0 651 566 A1 




14 



EP 0 651 566 A1 




15 



EP 0 651 566 A1 




EP 0 651 566 A1 




17 



EP 0 651 566 A1 



^ FPA |U 



-60 
FOV 












INERT! AL 
SYSTEM 














A/D 




SIGNAL 




DATA 


- 2.4 




PROCESSOR 




PROCESSOR 





COCOA 
WHEEL 




(Poor yWvJ 



OEWAR 

Figure \i Functional Block Diagram of a Conventional IRMWS 



18 



European Patent 
Office 



EUROPEAN SEARCH REPORT 



Application Number 

EP 94 11 5178 



DOCUMENTS CONSIDERED TO BE RELEVANT 




Category 


Citation of document with indication, where appropriate, 
of relevant passages 


Relevant 
to « 


CLASSIFICATION OF THE 
APPLICATION (Jat.CI.3 










X 

Y 
A 

A 
Y 
A 


1990 IEEE INTERNATIONAL SYMPOSIUM ON 
CIRCUITS AND SYSTEMS, 
vol. 4, 3 May 1990, NEW-ORLEANS (US) 
pages 3093 - 3096 

P.C. NGUYEN 'integrated gaas readout 
multiplexer and preprocessing electronics 
for infrared sensors 1 

* page 3093, left column, line 19 - line 
31 * 

* page 3094, right column, line 1 - line 
17 * 

US-A-5 055 683 (MC CRACKEN) 

* column 7, line 48 - line 61 * 

W0-A-86 06214 (IRVINE SENSORS CORPORATION) 

* page 7, line 1 - line 12 * 

* page 11, line 8 - line 12 * 


1,2 

6 
5 

1,5.6 
6 

1,5 


H04N5/33 


TECHNICAL FIELDS 
SEARCHED OU.CU) 


H04N 


The present search report has been drawn up for all claims 


Place af lean* Date of napUtloa of las tearca Bxawtaor 

THE HAGUE 6 January 1995 Bequet, T 


CATEGORY OF CITED DOCUMENTS T : theory or principle underlying the Invention 

E : earlier patent document, but published on, or 
X : particularly relevant 11 taken alone after the filing date 
Y : particularly relevant If combined with another D : document dted in the application 
document of the same category L : document dted for other reasons 

A : technological background „ ... ...... 

O : noo- written disdosure dk : member of the same patent family, corresponding 
P : intermediate document document 



